Benchmark-driven decision engine and kilo-auto/efficient

Summary

Adds a benchmark-driven model-routing pipeline behind a new hidden virtual model, kilo-auto/efficient: route each request to the cheapest model that is proven (by our own benchmarks) to be accurate enough for the request's difficulty.

Three moving parts:

• services/auto-routing-benchmark (new Cloudflare Worker): runs two deterministic benchmarks — classifier prompt replay via OpenRouter, and decider golden tasks through the real kilo CLI in a Cloudflare Container — writes normalized results to D1, and publishes a routing table (per-difficulty-tier ranked candidates) plus a classifier winner.
• services/auto-routing (existing worker): /decide classifies the request, derives a difficulty tier, and picks the cheapest above-threshold model from the routing table, with session-sticky decisions held in a Durable Object.
• apps/web (gateway): exposes kilo-auto/efficient, blocks on /decide with a 2s timeout, falls back to the balanced Qwen default, bills the classifier LLM cost to the requesting user, and adds an admin panel for the whole pipeline.

Shared classifier code (prompt, parsing, fallback, taxonomy, tier derivation, routing-table schema) moves into the new packages/auto-routing-contracts package, so the benchmark replays exactly the code the production worker executes.

Architecture

client request (model = kilo-auto/efficient)
        │
        ▼
┌────────────────────────────┐  POST /decide (2s timeout;      ┌─────────────────────────────────┐
│ apps/web gateway           │  null/timeout → balanced Qwen)  │ services/auto-routing           │
│ · resolves kilo-auto/*     │ ───────────────────────────────▶│ · classify request (LLM)        │
│ · applies pinned           │                                 │ · derive difficulty tier        │
│   reasoningEffort          │                                 │ · cheapest above-threshold pick │
│ · bills classifier cost    │                                 │ · sticky decision per           │
│ · admin panel              │                                 │   conversation (DO)             │
└──────────┬─────────────────┘                                 └──────────────┬──────────────────┘
           │ admin proxy +                                     routing table & classifier winner:
           │ 6h token mint                                     isolate cache 60s → KV 1h →
           │ (internal secret)                                 service binding origin
           ▼                                                                  ▼
┌───────────────────────────────────────────────────────────────────────────────────────────────┐
│ services/auto-routing-benchmark (new worker)                                                   │
│ · /admin/config /admin/runs /admin/routing-table /admin/classifier-winner /admin/debug-cli     │
│ · Queue (+DLQ) fans out per-model jobs                                                         │
│ · classifier bench: 72-case prompt replay via OpenRouter                                       │
│ · decider bench: 76 golden tasks via `kilo` CLI in a Cloudflare Container                      │
│ · D1 (drizzle, normalized): runs, case results, summaries, published routing tables            │
└───────────────────────────────────────────────────────────────────────────────────────────────┘

Benchmark worker (services/auto-routing-benchmark)

Classifier benchmark

Replays 72 normalized classifier inputs through OpenRouter using the exact production classifier code (@kilocode/auto-routing-contracts/classifier). Each output is graded per-field against a hand-labeled expectation via CLASSIFIER_FIELD_WEIGHTS (src/grading.ts): taskType 0.25, reasoningComplexity 0.20, contextComplexity 0.15, executionMode 0.15, subtaskType 0.10, requiresTools 0.10, riskLevel 0.05. Heuristic-fallback outputs score 0.

The winner (src/winner.ts) is the cheapest model meeting the run's accuracy threshold (most accurate one if none do). It feeds the worker's classifier-model resolution chain (below).

Decider benchmark

Runs 76 golden tasks per candidate model through the real kilo CLI (@kilocode/cli) inside a Cloudflare Container (container/Dockerfile + container/server.mjs, node:22-slim, standard-2). Grading is purely mechanical — exact / contains_all / regex / json_equal checks, no LLM judges; golden answers were hand-derived and mechanically re-verified where executable. Cases include genuinely agentic tasks performed with file/terminal tools inside the container (deterministic: no repo, no network).

Execution details:

• One queue message per (model, 10-case chunk); each chunk gets its own container instance (runId:model:chunk) so models/chunks never share state. CLI runs are serialized per instance (the CLI's sqlite state is not safe under concurrent first runs); a /warmup endpoint absorbs the one-time sqlite migration before the case loop.
• Each candidate's pinned reasoningEffort is forwarded as the CLI's --variant, so the benchmark measures the model exactly as it will be served.
• The CLI authenticates as a real Kilo user: the worker mints a short-lived token once per queue message via apps/web's internal endpoint (token only ever lives in a child-process env var, never logged or written to disk).
• Empty-output sessions (exit 0, no assistant text) are retried once, mirroring the production classifier's retry policy; costs of both attempts are summed.

Datasets

Both datasets cover all 18 (taskType, subtaskType) taxonomy pairs with at least 4 cases per pair — enforced by tests (src/datasets/*.test.ts). Decider cases each carry exactly one difficulty tier with at least 4 distinct task types per tier.

D1 schema (src/db-schema.ts)

Fully normalized, zero JSON blob columns, composite-PK-only access:

• benchmark_config + config_classifier_models + config_decider_models — admin config (incl. per-decider-model reasoning_effort).
• benchmark_runs — carries a config snapshot (min_accuracy, switch_cost_factor, max_concurrency, benchmark_user_id) taken at startRun time, so mid-run admin edits can't skew results. All job processing and publishing reads the snapshot, never live config.
• run_models — which models were enqueued vs. skipped, with the pinned reasoning_effort snapshot.
• case_results — per (run, model, case) score/latency/cost plus diagnostics (classifier fallback reason, CLI exit code/output prefix/event tail).
• model_summaries — per (run, model, tier) aggregates. Carried summaries: models with prior results are skipped on new runs (their latest summaries are copied in with carried=true), so re-runs only spend on new candidates; the admin can force a full re-run.
• routing_tables + routing_table_candidates — published tables, queryable history.

Single squashed baseline migration (migrations/0000_amused_shard.sql), applied by a predeploy script (wrangler d1 migrations apply --remote) which the CI deploy workflow now runs for any worker that defines one (.github/workflows/deploy-workers.yml).

Publishing

On run completion the worker builds the routing table from the run's own snapshot (src/routing-table-builder.ts): per tier, candidates are ranked best-bang-for-buck (above-threshold cheapest-first, below-threshold by accuracy). Models with zero graded cases or no cost signal in a tier are excluded; if any tier ends up empty the publish is skipped and the previous table stays live (schema enforces .min(1) per tier). Publishing only deletes the KV cache keys so the auto-routing worker repopulates from D1 on the next read.

Decision engine (services/auto-routing)

/decide (existing endpoint, now decision-capable):

1. Classifies the request (per-conversation classification cache in a Durable Object, keyed by classifier model + content hash).
2. Derives a difficulty tier (deriveDifficultyTier in contracts: reasoning complexity dominates at 2x weight; context, execution mode, and risk nudge borderline cases).
3. Picks from the routing table (src/decision-engine.ts): cheapest above-threshold candidate for the tier — unless the session has an incumbent.

Session stickiness: the conversation's Durable Object remembers the last served model. The incumbent is kept while it still meets the tier's accuracy threshold, unless the fresh pick is cheaper by more than the table's switchCostFactor. Rationale (commented in code): a model switch discards the provider's prompt cache, and rebuilding it costs full-price input tokens (4–10x cache-read rates) on a context that dominates agent-session spend — switching only pays off when recurring per-turn savings clearly exceed that one-time penalty. Sticky state trusts only real classifier output: heuristic fallbacks never re-anchor the session's model.

Routing table access: read-through chain — isolate-local 60s TTL cache → KV (1h TTL, shared AUTO_ROUTING_CONFIG namespace) → service binding to the benchmark worker's D1-backed /admin/routing-table. Corrupt KV values are treated as misses; origin failures degrade to null (no decision) rather than erroring the request.

Classifier model resolution (src/classifier-config.ts): admin KV override → benchmark winner (same KV read-through, derived on read) → built-in default google/gemini-2.5-flash-lite. A benchmark-origin failure never discards a healthy override.

Gateway (apps/web)

• kilo-auto/efficient (src/lib/ai-gateway/auto-model/index.ts): hidden virtual model (excluded from /models, usable by id) with the same catalog properties as balanced — intended to eventually replace it, hidden while validated on Kilo team traffic.
• Resolution (auto-model/resolution.ts + auto-routing-decision.ts): blocks on /decide with a 2s timeout; on a decision, serves the decided model and applies its pinned reasoningEffort so it runs under the same conditions the benchmark measured. On null/timeout/error, serves BALANCED_QWEN_MODEL — an efficient request never degrades below balanced.
• Billing: the classifier LLM cost returned by /decide is billed to the requesting user as a separate microdollar usage row (requested_model: kilo-auto/efficient), so routing overhead is visible and attributed rather than absorbed.
• Admin panel (admin/auto-routing/BenchmarksSection.tsx, proxied through admin API routes with the internal secret): config editor (classifier/decider model lists, per-decider reasoningEffort, minAccuracy, switchCostFactor, maxConcurrency, benchmarkUserId), run triggers with a force-rerun toggle, run history, and the live published routing table.
• Config-save validation (admin/api/auto-routing/benchmark-config/route.ts): every decider model must be servable on all gateway chat API kinds (chat_completions, responses, messages) by the provider the gateway would route it to — the routing table deliberately carries no per-protocol metadata, so this invariant is enforced at write time.
• Token mint (api/internal/auto-routing-benchmark/token/route.ts): POST gated by INTERNAL_API_SECRET; mints a 6h full user API token (tokenSource: auto-routing-benchmark) for the decider CLI's identity/billing.

Design properties

• No fabricated data anywhere. There is no default routing table: /decide returns null decisions until a benchmark publishes one, and the gateway serves balanced fallbacks. There is no default benchmark config: runs refuse to start until an admin saves one (and decider runs additionally fail fast without a benchmarkUserId).
• Deterministic, reproducible grading. Mechanical checks only; run-level config snapshots; routing tables built from the run's snapshot, not live config.
• Cheap iteration. Carried summaries mean adding one candidate model re-benchmarks only that model; config-only changes (model removed, threshold tweaked) republish instantly with zero spend.
• Graceful degradation at every layer. Corrupt KV → miss; origin failure → previous behavior; classifier failure → no decision → balanced fallback; publish failure → previous table stays live.

Infrastructure

• D1 auto-routing-benchmark in region EEUR, primary in Frankfurt (colo FRA — next to the backend; verified via wrangler d1 info).
• Queue auto-routing-benchmark-jobs (max_concurrency 4, max_retries 2) + DLQ auto-routing-benchmark-dlq.
• Container app auto-routing-benchmark-runner (standard-2, max 40 instances), image built and pushed by wrangler deploy.
• Service binding auto-routing → auto-routing-benchmark; shared KV namespace AUTO_ROUTING_CONFIG.
• Both workers already run this branch's code; the D1 database is empty pending admin setup.

Post-merge deploy / cutover checklist

1. Merge → Vercel ships the gateway side (kilo-auto/efficient, admin panel, token mint).
2. First post-merge worker deploy runs the D1 migration via the new CI predeploy hook — CI's CLOUDFLARE_API_TOKEN needs D1 edit permission (the deploy will surface it if missing).
3. Admin saves a benchmark config: benchmarkUserId is required for decider runs (consider a dedicated service account — its account is billed for CLI usage); suggested switchCostFactor starting value: 3.
4. Trigger a classifier run and a decider run from the admin panel.
5. Clear the leftover classifier_model KV override (currently set to flash-lite) if the benchmark winner should drive classifier selection.

Reviewer notes

• The exact decider check also accepts the last non-empty output line (src/grading.ts): agent harnesses sometimes prepend preamble despite instructions; wrong answers fail either way.
• @kilocode/cli@latest is resolved at image build time, i.e. each deploy pins whatever was latest then; re-deploy to pick up a newer CLI.
• The token-mint endpoint is gated by INTERNAL_API_SECRET and can mint for any user id; scoping it to the configured benchmark user is a reasonable follow-up.
• The decider benchmark exercises models through chat_completions only (the CLI's path). Config-save validation guarantees candidates are servable on all three chat API kinds, but accuracy is only measured on one.